Direct data loading of middleware-generated records

ABSTRACT

A computer-implemented method is presented for reducing data loading overhead of middleware to facilitate direct data loading to a database management system (DBMS). The method includes, while data loading, sending external-writes to a memory-based DBMS if corresponding internal-writes are for vertices, exporting all the external-writes to a disk-based DBMS as an export file, and sending an external-read for vertices to an in-memory DBMS if the middleware requests data. The method further includes, at the end of data loading, generating files for raw data of the disk-based DBMS from the export file and sending the generated raw files to the disk-based DBMS.

BACKGROUND

The present invention relates generally to database management systems,and more specifically, to direct data loading of middleware-generatedrecords to database management systems.

The ability to act quickly and decisively in today's increasinglycompetitive marketplace is important to the success of organizations.The volume of information that is available to corporations is rapidlyincreasing and frequently overwhelming. Those organizations that willeffectively and efficiently manage these massive volumes of data, anduse the information to make business decisions, will realize acompetitive advantage in the marketplace. Such competitive advantage canbe achieved by using a Database Management System (DBMS), which storeslarge volumes of data to support diverse workloads and heterogeneousapplications. The DBMS is beneficial to business transaction processingand decision making, and can incorporate strategies that promote keepingthe data highly available.

The DBMS is a database program that uses a standard method ofcataloging, retrieving, and running queries on data. The DBMS managesincoming data, organizes the data, and provides ways the data can bemodified or extracted by users or other programs. The DBMS provides aquery language and report writer that allows users to interactivelyinterrogate the relational database. These essential components giveusers access to all management information as needed. The DBMS appliesentered text to the database as criteria to identify and report recordsin the database that meet the criteria. Entering the text into thefields of the DBMS requires the user to have an understanding of theDBMS and how the DBMS represents the data in the database. For example,for a given search term, the user must know which field in the DBMS isappropriate for searching that term. For a search query with multiplesearch terms, the user must be familiar with multiple fields in theDBMS, and how those fields will interact to limit or otherwise define asearch. Also, the user must know the hierarchical structure betweendatabase tables and the keys for linking the tables together.

A common goal of DBMSs is to provide high performance in terms oftransaction throughput and transaction latency to minimize hardwarecost, wait time and increase the number of transactions per unit oftime. Even with large investments in hardware, achieving the desiredperformance is often expensive and sometimes not possible. Anothercommon goal of DBMSs is to reduce complexity with respect to theapplication development process and thus save time and money, as well asreduce the risk of errors.

However, conventional DBMSs are built around the assumption that dataloading is a “one-time deal.” Data loading is considered an offlineprocess out of the critical path, with the user defining a schema andloading the majority of the data in “one go” before submitting anyqueries. When this architectural design assumption is combined with theexplosive data growth, the result is the emergence of data loading as abottleneck in the data analysis pipeline of DBMSs.

Nevertheless, some analytics need data loading to DBMS in advance.Middleware can be beneficial if a DBMS provides a simple applicationprogramming interface (API) for analytics. However, middleware can alsopresent overheads when data loading takes place. Thus, data loading inDBMSs can cause bottlenecks. Approaches are thus necessary to reducedata loading in DBMS.

SUMMARY

In accordance with an embodiment, a method is provided for reducing dataloading overhead of middleware to facilitate direct data loading to adatabase management system (DBMS). The method includes, while dataloading, sending external-writes to a memory-based DBMS if correspondinginternal-writes are for vertices, exporting all the external-writes to adisk-based DBMS as an export file, and sending an external-read forvertices to an in-memory DBMS if the middleware requires data. Themethod further includes, at the end of data loading, generating filesfor raw data of the disk-based DBMS from the export file and sending thegenerated raw files to the disk-based DBMS.

In accordance with another embodiment, a system is provided for reducingdata loading overhead of middleware to facilitate direct data loading toa database management system (DBMS). The system includes a memory andone or more processors in communication with the memory configured to,while data loading, send external-writes to a memory-based DBMS ifcorresponding internal-writes are for vertices, export all theexternal-writes to a disk-based DBMS as an export file, and send anexternal-read for vertices to an in-memory DBMS if the middlewarerequires data. The system, at the end of data loading, further generatesfiles for raw data of the disk-based DBMS from the export file and sendsthe generated raw files to the disk-based DBMS.

In accordance with yet another embodiment, a non-transitorycomputer-readable storage medium comprising a computer-readable programfor reducing data loading overhead of middleware to facilitate directdata loading to a database management system (DBMS) is presented. Thenon-transitory computer-readable storage medium performs the steps of,while data loading, sending external-writes to a memory-based DBMS ifcorresponding internal-writes are for vertices, exporting all theexternal-writes to a disk-based DBMS as an export file, and sending anexternal-read for vertices to an in-memory DBMS if the middlewarerequires data. The non-transitory computer-readable storage mediumperforms the steps of, at the end of data loading, generating files forraw data of the disk-based DBMS from the export file and sending thegenerated raw files to the disk-based DBMS.

In accordance with an embodiment, a method is provided for reducing dataloading overhead of middleware to facilitate direct data loading to adatabase management system (DBMS). The method includes receiving aninternal-write in an export extension of the middleware, determiningwhether the internal-write is for vertices, sending the internal-writeto an in-memory DBMS when the internal write is for vertices, andappending the internal-write to a recovery file.

In accordance with another embodiment, a method is provided for reducingdata loading overhead of middleware to facilitate direct data loading toa database management system (DBMS). The method includes receiving aninternal-read in an export extension of the middleware, sending theinternal-read to an in-memory DBMS to receive a result, determiningwhether the result includes a record, and if the result is free of arecord, sending the internal-read to a disk-based DBMS.

The advantages of the present invention include reducing data loadingvia middleware. The advantages of the present invention further includemore efficient central processing unit (CPU) utilization and moreefficient input/output (I/O) bandwidth utilization. Data loading is anupfront investment that DBMSs have to undertake in order to be able tosupport efficient query execution. Given the amount of data gathered byapplications today, it is important to minimize the overhead of dataloading to prevent data loading from becoming a bottleneck in the dataanalytics pipeline. This results in a higher storage capacity, fasterprocessing, and better transfer speed of the unstructured data. Furtheradvantages include higher quality, reduced cost, clearer scope, fasterperformance, fewer application errors, and fewer data errors.

In one preferred aspect, while data loading, the sending of theexternal-writes step further includes exporting the internal-writes as arecovery file.

In another preferred aspect, at a start of data loading, if the recoveryfile exists, generate the external-writes to the memory-based DBMS fromthe recovery file and send the generated external-writes to thememory-based DBMS.

In yet another preferred aspect, while data loading, the sending of theexternal-read step further includes sending the external-read to thedisk-based DBMS if the in-memory DBMS does not fetch any.

In yet another preferred aspect, an export extension of the middlewaresupports both disk-based DBMS and memory-based DBMS.

In yet another preferred aspect, an export extension of the middlewareprocesses the internal-writes and internal reads.

In yet another preferred aspect, if internal-writes are not forvertices, the internal-writes are appended to the export file.

It should be noted that the exemplary embodiments are described withreference to different subject-matters. In particular, some embodimentsare described with reference to method type claims whereas otherembodiments have been described with reference to apparatus type claims.However, a person skilled in the art will gather from the above and thefollowing description that, unless otherwise notified, in addition toany combination of features belonging to one type of subject-matter,also any combination between features relating to differentsubject-matters, in particular, between features of the method typeclaims, and features of the apparatus type claims, is considered as tobe described within this document.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will provide details in the following description ofpreferred embodiments with reference to the following figures wherein:

FIG. 1 is an exemplary middleware architecture for direct data loadingof middleware-generated records to database management systems (DBMS),in accordance with an embodiment of the present invention;

FIG. 2 is an exemplary flowchart illustrating a methodology whenreceiving an internal-write, in accordance with an embodiment of thepresent invention;

FIG. 3 is an exemplary flowchart illustrating a methodology whenreceiving an internal-read, in accordance with an embodiment of thepresent invention;

FIG. 4 is an exemplary flowchart illustrating a methodology for startinginitialization of export extension, in accordance with an embodiment ofthe present invention;

FIG. 5 is an exemplary middleware diagram, in accordance with anembodiment of the present invention;

FIG. 6 is an exemplary diagram illustrating a request/response cycle, inaccordance with an embodiment of the present invention;

FIG. 7 is an exemplary JanusGraph architecture, in accordance with anembodiment of the present invention;

FIG. 8 is an exemplary processing system, in accordance with anembodiment of the present invention;

FIG. 9 is a block/flow diagram of an exemplary cloud computingenvironment, in accordance with an embodiment of the present invention;

FIG. 10 is a schematic diagram of exemplary abstraction model layers, inaccordance with an embodiment of the present invention;

FIG. 11 is a block/flow diagram of a method for applying the middlewarearchitecture of FIG. 1 with Internet of Things (IoT)systems/devices/infrastructure, in accordance with an embodiment of thepresent invention;

FIG. 12 is a block/flow diagram of exemplary IoT sensors used to collectdata/information related to the middleware architecture of FIG. 1, inaccordance with an embodiment of the present invention; and

FIG. 13 is a block/flow diagram of an exemplary method for direct dataloading of middleware-generated records to DBMS, in accordance with anembodiment of the present invention.

Throughout the drawings, same or similar reference numerals representthe same or similar elements.

DETAILED DESCRIPTION

Embodiments in accordance with the present invention provide methods anddevices for employing direct data loading of middleware-generatedrecords to database management systems (DBMS). DBMS is a collection ofinter-related data and set of programs to store and access those data inan easy and effective manner. Database systems are developed for largeamounts of data. When dealing with large amounts of data, there are twothings that need optimization, storage of data and retrieval of data.Data load is the process that involves taking transformed data andloading the transformed data where users can access such transformeddata. Data loading has been considered a “one-time deal,” that is anoffline process out of the critical path of query execution. Thearchitecture of DBMS is aligned with this assumption. Nevertheless, therate in which data is produced and gathered nowadays has nullified the“one-off” assumption, and has turned data loading into a bottleneck ofthe data analysis pipeline.

The exemplary embodiments of the present invention advantageously assumemiddleware converts an internal representation of an update(internal-write) to one or more commands of updates (external-write) forDBMS. Moreover, an internal-write for vertices can be advantageouslyidentified in data loading, middleware can advantageously supportdisk-based and memory-based DBMS, and disk-based DBMS advantageouslyprovides a way to directly add files for its raw data.

The exemplary embodiments of the present invention advantageously employan extension of middleware that manages both disk-based and memory-basedDBMS. While data loading, the extension sends external-writes to thememory-based DBMS if corresponding internal-writes are for vertices,exports the internal-writes as a file (recovery-file), exports all theexternal-write to disk-based DBMS as a file (export-file), and sends anexternal-read for vertices to the in-memory DBMS if middlewarerequires/requests data. In another option, the extension sends anexternal-read to the disk-based DBMS if the in-memory DBMS does notfetch any. At the end of data loading, the extension or the othercomponent advantageously generates files for raw data of the disk-basedDBMS from the export-file and sends them to the disk-based DBMS. Inanother embodiment, at the beginning of data loading, if a recovery-fileexits, the extension advantageously generates external-writes to thememory-based DBMS from the file and sends them to the memory-based DBMS.

It is to be understood that the present invention will be described interms of a given illustrative architecture; however, otherarchitectures, structures, substrate materials and process features andsteps/blocks can be varied within the scope of the present invention. Itshould be noted that certain features cannot be shown in all figures forthe sake of clarity. This is not intended to be interpreted as alimitation of any particular embodiment, or illustration, or scope ofthe claims.

FIG. 1 is an exemplary middleware architecture for direct data loadingof middleware-generated records to database management systems (DBMS),in accordance with an embodiment of the present invention.

The middleware architecture 10 includes a data loading application 12where vertices and edges of graphs are fed into TinkerPop™ API 13. Agraph is a structure composed of vertices and edges. Both vertices andedges can have an arbitrary number of key/value-pairs called properties.Vertices denote discrete objects such as a person, a place, or an event.Edges denote relationships between vertices. For instance, a personmight know another person, have been involved in an event, and/or haverecently been at a particular place. Properties express non-relationalinformation about the vertices and edges. Example properties include avertex having a name and an age, and an edge having a timestamp and/or aweight. Together, the aforementioned graph is known as a property graphand it is the foundational data structure of Apache TinkerPop™. ApacheTinkerPop™ is a graph computing framework for both graph databases(OLTP) and graph analytic systems (OLAP).

The TinkerPop™ API 13 can be integrated with JanusGraph 20. JanusGraph20 is a scalable graph database optimized for storing and queryinggraphs including hundreds of billions of vertices and edges distributedacross a multi-machine cluster. JanusGraph 20 includes a JanusGraph Core22, a first storage manager SPI 23, and a second storage manager SPI 40.The JanusGraph Core 22 provides internal-reads and internal-writes tothe storage manager SPI 23. The storage manager 23 advantageouslyincludes an export extension 24. The export extension 24 receivesinternal-writes and internal-reads and processes them. The exportextension 24 is a component that supports SPIs 23, 40. The exportextension 24 coordinates multiple extensions that support SPIs 23, 40with functions of pre-load and export data. Since the export extension24 uses only SPIs 23, 40 to access other extensions, no modification toJanusGraph 20 is necessary.

The export extension 24 receives recovery-files 30, and while dataloading the extension, sends external-writes to the in-memory-based DBMS44 if its corresponding internal-writes are for vertices while dataloading. If the internal-write is for vertices, then the internal-writeis sent to the in memory DBMS extension 44. In a first option, when theinternal-write is for vertices, the internal-write is advantageouslyexported as a recovery file 32. In a second option, when theinternal-write is not for vertices, the method advantageously exportsall the external-writes to the disk-based DBMS extension 42 as an exportfile 34, and sends an external-read, via a disk-based DBMS driver 46,for vertices to the in-memory DBMS if the middleware requires/requestsit.

The in-memory database 44 is based on a database management system thatstores its data collections directly in the working memory of one ormore computers. Using random access memory (RAM) has a key advantage inthat in-memory databases have significantly faster access speeds. Storeddata is then available very quickly when needed. The biggest advantageof using in-memory databases is the significantly higher access speedsresulting from the use of RAM. This also leads to a quicker dataanalysis. However, it's not only the reduced fetch time that optimizesdata analysis. In-memory DBs advantageously make the evaluation ofstructured and unstructured data possible from any system.

At the end of data loading, the extension 24 advantageously generatesfiles 52 for raw data of the disk-based DBMS 56 from the export-file 34.The extension 24 advantageously sends the raw files 52 to the disk-basedDBMS 56 via a disk-based DBMS tool 54 thereby providing a way todirectly add files for its raw data using disk-based DBMS. The converter50 advantageously converts the export file 34 to raw files 52.

FIG. 2 is an exemplary flowchart illustrating a methodology whenreceiving an internal-write, in accordance with an embodiment of thepresent invention.

At block 102, an internal-write is received.

At block 104, it is determined whether the internal-write relates tovertices.

If NO, the process proceeds to block 110. If YES, the process proceedsto block 106.

At block 106, the internal-write is sent to the in-memory DBMSextension.

At block 108, the internal-write is appended to the recovery-file, whenthe internal-write is for vertices.

At block 110, the internal-write is appended to the export-file, whenthe internal-write is not for vertices.

FIG. 3 is an exemplary flowchart illustrating a methodology whenreceiving an internal-read, in accordance with an embodiment of thepresent invention.

At block 152, an internal-read is received.

At block 154, the internal-read is sent to the in-memory DBMS extensionand the result is received.

At block 156, it is determined whether the result includes a record.

If YES, the process proceeds to block 160. If NO, the process proceedsto block 158.

At block 158, the internal-read is sent to the disk-based DBMS extensionand the result is received.

At block 160, the result is returned to the caller.

FIG. 4 is an exemplary flowchart illustrating a methodology for startinginitialization of export extension, in accordance with an embodiment ofthe present invention.

Regarding the appending of the internal-write to the recovery file orblock 108, the following takes place.

At block 182, the initialization of the export extension is commenced.

At block 184, it is determined whether the recovery-file exists.

If YES, the process proceeds to block 190. If NO, the process proceedsto block 186.

At block 186, all the internal-writes are sent to the in-memory DBMSextension.

At block 188, all the internal writes are copied to the newrecovery-file.

At block 190, the initialization of the export extension is ended.

FIG. 5 is an exemplary middleware diagram, in accordance with anembodiment of the present invention.

Middleware can be described as the glue that combines two separate,already existing software programs. Generally, middleware can be avariety of different, specifically designed software. Middleware is atype of software that makes it easier for software developers toimplement communication and input/output, between the two programs, sosoftware developers can focus on the specific purpose of theirapplication. Acting as almost an extension to the existing operatingsystems, middleware helps to integrate software between applications andservices.

Middleware can be used to connect any two pieces of software. Middlewareworks by allowing data to be passed between the two. One example of theuse of middleware is when middleware is used to connect a databasesystem with a web server. This allows the user to request data from thedatabase using forms displayed on a web browser. Middleware is softwarethat provides common services and capabilities to applications outsideof what's offered by the operating system. Data management, applicationservices, messaging, authentication, and API management are all commonlyhandled by middleware. Moreover, middleware helps developers buildapplications more efficiently. Middleware acts like the connectivetissue between applications, data, and users. For organizations withmulti-cloud and containerized environments, middleware can make itcost-effective to develop and run applications at scale.

In middleware diagram 200, middleware 210 advantageously connectssystems of engagement 220 with systems of record 240. The systems ofengagement 220 can include, but are not limited to, public cloud systems222, mobile systems 224, internet-of-things (IoT) systems 226, socialmedia systems 228, affiliates 230, and websites 232. The systems ofrecord can include, but are not limited to, private clouds 242, customerrelationship management (CRM) 244, servers 246, databases 248,back-office processes 250, and application programming interface (API)services 252.

FIGS. 11 and 12 below will illustrate one practical application relatedto IoT systems 226 advantageously employing middleware 210 to connectwith systems of record 240.

FIG. 6 is an exemplary diagram illustrating a request/response cycle, inaccordance with an embodiment of the present invention.

In diagram 260 the requests 262 and the responses 264 for theapplication 270 are handled by the middleware 272. A “middleware” 272 isa function that works with every request 262 before it is processed byany specific path operation, and also with every response beforereturning it. Middleware 272 takes each request 262 that comes to theapplication 270. Middleware 272 can then do something to that request262 or run any needed code. Then the middleware 272 passes the request262 to be processed by the rest of the application 270 (by some pathoperation). Middleware 272 then takes the response 264 generated by theapplication 270 (by some path operation). Middleware 272 can dosomething to that response 264 or run any needed code. Then themiddleware 272 returns the response 264.

FIG. 7 is an exemplary JanusGraph architecture, in accordance with anembodiment of the present invention.

JanusGraph is a scalable transactional property graph database. Aproperty graph is a mapping between entities (which are referred to asvertices) by their relationships (which are referred to as edges).Property graph queries can traverse multiple edges and vertices to helpilluminate the relationships between entities. Thus, JanusGraph is ascalable graph database that has the advantage of being optimized forstoring and querying graphs including hundreds of billions of verticesand edges distributed across a multi-machine cluster.

The JanusGraph architecture 300 includes applications 302 fed into theJanusGraph 310 and the Gremlin Graph Computer 330. The Gremlin GraphComputer 330 communicates with big data platform 335.

The storage backend 340 is pluggable and supports at least Cassandra,HBase, BerkeleyDB, Google BigTable and an in-memory storage option. Thestorage backend 340 is where the data is actually stored. Given itsflexibility to work with numerous database engines, the storage backend340 has the advantage of allowing a user to pick an option that mightalready have been deployed or possessed expertise in by the user. Thereis only one storage backend 340.

Next for the external index backend 350, at least Elasticsearch, Solr,and Lucene are supported. The external index backend 350 is optional,but necessary for indexing on multiple properties, full text and stringsearches, and geo-mapping. Once again there is only one external indexbackend 350.

After the storage backends 340, 350, the TinkerPop™ API 321 in Gremlinbox 310 represents how a user can interact with the graph. It's commonlyknown as the Gremlin Console and is an example of an application thatinvokes the TinkerPop™ API 321. This is the command line interface usedto interact with JanusGraph.

Finally, 310 represents the JanusGraph Server. This piece runs with ascript named gremlin_server.sh. Gremlin Server is a part of the ApacheTinkerPop™ project. JanusGraph essentially has the advantage of actingas a plugin for Gremlin Server and tells the Gremlin Server how, andwhere, to store graph data. The JanusGraph Server 310 advantageouslyincludes management API 320, internal API layer 322, database layer 324,storage and index interface layer 326, as well as an OLAP input/output(I/O) interface 328.

Gremlin is Titan's query language used to retrieve data from and modifydata in the graph. Gremlin is a path-oriented language which succinctlyexpresses complex graph traversals and mutation operations. Gremlin is afunctional language whereby traversal operators are chained together toform path-like expressions. Gremlin works for both online transactionprocessing (OLTP)-based graph databases as well as online analyticalprocessing (OLAP)-based graph processors. Gremlin's automata andfunctional language foundation enable Gremlin to naturally supportimperative and declarative querying, host language agnosticism,user-defined domain specific languages, an extensiblecompiler/optimizer, single- and multi-machine execution models, andhybrid depth- and breadth-first evaluation.

In summary, the exemplary embodiments of the present inventionadvantageously assume middleware converts an internal representation ofan update (internal-write) to one or more commands of updates(external-write) for DBMS. Moreover, an internal-write for vertices canbe advantageously identified in data loading, middleware canadvantageously support disk-based and memory-based DBMS, and disk-basedDBMS advantageously provides a way to directly add files for its rawdata. The exemplary embodiments of the present invention advantageouslyemploy an extension of middleware that manages both disk-based andmemory-based DBMS. While data loading, the extension sendsexternal-writes to the memory-based DBMS if its correspondinginternal-writes are for vertices, exports the internal-writes as a file(recovery-file), exports all the external-write to disk-based DBMS as afile (export-file), and sends an external-read for vertices to thein-memory DBMS if middleware requires a data. In another option, theextension sends an external-read to the disk-based DBMS if the in-memoryDBMS does not fetch any. At the end of data loading, the extension orthe other component generates files for raw data of the disk-based DBMSfrom the export-file and sends them to the disk-based DBMS. In anotherembodiment, at the beginning of data loading, if a recovery-file exits,the extension generates external-writes to the memory-based DBMS fromthe file and sends them to the memory-based DBMS.

The advantages of the present invention include reducing data loadingvia middleware. The advantages of the present invention further includemore efficient CPU utilization and more efficient I/O bandwidthutilization. Data loading is an upfront investment that DBMSs have toundertake in order to be able to support efficient query execution.Given the amount of data gathered by applications, it is important tominimize the overhead of data loading to prevent data loading frombecoming a bottleneck in the data analytics pipeline. This results in ahigher storage capacity, faster processing, and better transfer speed ofthe unstructured data. Further advantages include higher quality,reduced cost, clearer scope, faster performance, fewer applicationerrors, and fewer data errors.

FIG. 8 is an exemplary processing system for handling streamingalgorithms, in accordance with embodiments of the present invention.

Referring now to FIG. 8, this figure shows a hardware configuration ofcomputing system 600 according to an embodiment of the presentinvention. As seen, this hardware configuration has at least oneprocessor or central processing unit (CPU) 611. The CPUs 611 areinterconnected via a system bus 612 to a random access memory (RAM) 614,read-only memory (ROM) 616, input/output (I/O) adapter 618 (forconnecting peripheral devices such as disk units 621 and tape drives 640to the bus 612), user interface adapter 622 (for connecting a keyboard624, mouse 626, speaker 628, microphone 632, and/or other user interfacedevice to the bus 612), a communications adapter 634 for connecting thesystem 600 to a data processing network, the Internet, an Intranet, alocal area network (LAN), etc., and a display adapter 636 for connectingthe bus 612 to a display device 638 and/or printer 639 (e.g., a digitalprinter or the like).

FIG. 9 is a block/flow diagram of an exemplary cloud computingenvironment, in accordance with an embodiment of the present invention.

FIG. 9 is a block/flow diagram of an exemplary cloud computingenvironment, in accordance with an embodiment of the present invention.

It is to be understood that although this invention includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model can includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but can be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It can be managed by the organization or a third party andcan exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It can be managed by the organizations or a third partyand can exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 9, illustrative cloud computing environment 750 isdepicted for enabling use cases of the present invention. As shown,cloud computing environment 750 includes one or more cloud computingnodes 710 with which local computing devices used by cloud consumers,such as, for example, personal digital assistant (PDA) or cellulartelephone 754A, desktop computer 754B, laptop computer 754C, and/orautomobile computer system 754N can communicate. Nodes 710 cancommunicate with one another. They can be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 750 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 754A-Nshown in FIG. 9 are intended to be illustrative only and that computingnodes 710 and cloud computing environment 750 can communicate with anytype of computerized device over any type of network and/or networkaddressable connection (e.g., using a web browser).

FIG. 10 is a schematic diagram of exemplary abstraction model layers, inaccordance with an embodiment of the present invention. It should beunderstood in advance that the components, layers, and functions shownin FIG. 10 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Hardware and software layer 860 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 861;RISC (Reduced Instruction Set Computer) architecture based servers 862;servers 863; blade servers 864; storage devices 865; and networks andnetworking components 866. In some embodiments, software componentsinclude network application server software 867 and database software868.

Virtualization layer 870 provides an abstraction layer from which thefollowing examples of virtual entities can be provided: virtual servers871; virtual storage 872; virtual networks 873, including virtualprivate networks; virtual applications and operating systems 874; andvirtual clients 875.

In one example, management layer 880 can provide the functions describedbelow. Resource provisioning 881 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 882provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources can include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 883 provides access to the cloud computing environment forconsumers and system administrators. Service level management 884provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 885 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 890 provides examples of functionality for which thecloud computing environment can be utilized. Examples of workloads andfunctions which can be provided from this layer include: mapping andnavigation 891; software development and lifecycle management 892;virtual classroom education delivery 893; data analytics processing 894;transaction processing 895; and direct data loading 896.

FIG. 11 is a block/flow diagram of a method for applying the middlewarearchitecture of FIG. 1 with Internet of Things (IoT)systems/devices/infrastructure, in accordance with an embodiment of thepresent invention.

According to some embodiments of the invention, a network is implementedusing an IoT methodology. For example, direct data loading 902 can beincorporated, e.g., into wearable, implantable, or ingestible electronicdevices and Internet of Things (IoT) sensors. The wearable, implantable,or ingestible devices can include at least health and wellnessmonitoring devices, as well as fitness devices. The wearable,implantable, or ingestible devices can further include at leastimplantable devices, smart watches, head-mounted devices, security andprevention devices, and gaming and lifestyle devices. The IoT sensorscan be incorporated into at least home automation applications,automotive applications, user interface applications, lifestyle and/orentertainment applications, city and/or infrastructure applications,toys, healthcare, fitness, retail tags and/or trackers, platforms andcomponents, etc. The direct data loading 902 described herein can beincorporated into any type of electronic devices for any type of use orapplication or operation.

IoT systems allow users to achieve deeper automation, analysis, andintegration within a system. IoT improves the reach of these areas andtheir accuracy. IoT utilizes existing and emerging technology forsensing, networking, and robotics. Features of IoT include artificialintelligence, connectivity, sensors, active engagement, and small deviceuse. In various embodiments, the direct data loading 902 of the presentinvention can be incorporated into a variety of different devices and/orsystems. For example, the direct data loading 902 can be incorporatedinto wearable or portable electronic devices 904. Wearable/portableelectronic devices 904 can include implantable devices 940, such assmart clothing 943. Wearable/portable devices 904 can include smartwatches 942, as well as smart jewelry 945. Wearable/portable devices 904can further include fitness monitoring devices 944, health and wellnessmonitoring devices 946, head-mounted devices 948 (e.g., smart glasses949), security and prevention systems 950, gaming and lifestyle devices952, smart phones/tablets 954, media players 956, and/orcomputers/computing devices 958.

The direct data loading 902 of the present invention can be furtherincorporated into Internet of Thing (IoT) sensors 906 for variousapplications, such as home automation 920, automotive 922, userinterface 924, lifestyle and/or entertainment 926, city and/orinfrastructure 928, retail 910, tags and/or trackers 912, platform andcomponents 914, toys 930, and/or healthcare 932, as well as fitness 934.The IoT sensors 906 can employ the direct data loading 902. Of course,one skilled in the art can contemplate incorporating such direct dataloading 902 into any type of electronic devices for any types ofapplications, not limited to the ones described herein.

FIG. 12 is a block/flow diagram of exemplary IoT sensors used to collectdata/information related to the middleware architecture of FIG. 1, inaccordance with an embodiment of the present invention.

IoT loses its distinction without sensors. IoT sensors act as defininginstruments which transform IoT from a standard passive network ofdevices into an active system capable of real-world integration.

The IoT sensors 906 can employ the direct data loading 902 to transmitinformation/data, continuously and in real-time, via a network 908, toany type of distributed system. Exemplary IoT sensors 906 can include,but are not limited to, position/presence/proximity sensors 1002,motion/velocity sensors 1004, displacement sensors 1006, such asacceleration/tilt sensors 1007, temperature sensors 1008,humidity/moisture sensors 1010, as well as flow sensors 1011,acoustic/sound/vibration sensors 1012, chemical/gas sensors 1014,force/load/torque/strain/pressure sensors 1016, and/or electric/magneticsensors 1018. One skilled in the art can contemplate using anycombination of such sensors to collect data/information of thedistributed system for further processing. One skilled in the art cancontemplate using other types of IoT sensors, such as, but not limitedto, magnetometers, gyroscopes, image sensors, light sensors, radiofrequency identification (RFID) sensors, and/or micro flow sensors. IoTsensors can also include energy modules, power management modules, RFmodules, and sensing modules. RF modules manage communications throughtheir signal processing, WiFi, ZigBee®, Bluetooth®, radio transceiver,duplexer, etc.

FIG. 13 is a block/flow diagram of an exemplary method for learningrelationships between multiple event types, in accordance with anembodiment of the present invention.

At block 1302, while data loading, send external-writes to amemory-based database management system (DBMS) if correspondinginternal-writes are for vertices, export all the external-writes to adisk-based DBMS as a file (export-file), and transmit an external-readfor vertices to an in-memory DBMS if the middleware requires data.

At block 1304, at the end of data loading, generate files for raw dataof the disk-based DBMS from the export-file and transmit the generatedfiles to the disk-based DBMS.

The present invention can be a system, a method, and/or a computerprogram product. The computer program product can include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium can be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network can includecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention can be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions can execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer can be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection can be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) can execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions can be provided to at leastone processor of a general purpose computer, special purpose computer,or other programmable data processing apparatus to produce a machine,such that the instructions, which execute via the processor of thecomputer or other programmable data processing apparatus, create meansfor implementing the functions/acts specified in the flowchart and/orblock diagram block or blocks or modules. These computer readableprogram instructions can also be stored in a computer readable storagemedium that can direct a computer, a programmable data processingapparatus, and/or other devices to function in a particular manner, suchthat the computer readable storage medium having instructions storedtherein includes an article of manufacture including instructions whichimplement aspects of the function/act specified in the flowchart and/orblock diagram block or blocks or modules.

The computer readable program instructions can also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational blocks/steps to be performed on thecomputer, other programmable apparatus or other device to produce acomputer implemented process, such that the instructions which executeon the computer, other programmable apparatus, or other device implementthe functions/acts specified in the flowchart and/or block diagram blockor blocks or modules.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams can represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks can occur out of theorder noted in the figures. For example, two blocks shown in successioncan, in fact, be executed substantially concurrently, or the blocks cansometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Reference in the specification to “one embodiment” or “an embodiment” ofthe present principles, as well as other variations thereof, means thata particular feature, structure, characteristic, and so forth describedin connection with the embodiment is included in at least one embodimentof the present principles. Thus, the appearances of the phrase “in oneembodiment” or “in an embodiment”, as well any other variations,appearing in various places throughout the specification are notnecessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following “/”,“and/or”, and “at least one of”, for example, in the cases of “A/B”, “Aand/or B” and “at least one of A and B”, is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of both options (A andB). As a further example, in the cases of “A, B, and/or C” and “at leastone of A, B, and C”, such phrasing is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of the third listedoption (C) only, or the selection of the first and the second listedoptions (A and B) only, or the selection of the first and third listedoptions (A and C) only, or the selection of the second and third listedoptions (B and C) only, or the selection of all three options (A and Band C). This can be extended, as readily apparent by one of ordinaryskill in this and related arts, for as many items listed.

Having described preferred embodiments of a system and method forreducing data loading overhead of middleware to facilitate direct dataloading to a database management system (DBMS) (which are intended to beillustrative and not limiting), it is noted that modifications andvariations can be made by persons skilled in the art in light of theabove teachings. It is therefore to be understood that changes may bemade in the particular embodiments described which are within the scopeof the invention as outlined by the appended claims. Having thusdescribed aspects of the invention, with the details and particularityrequired by the patent laws, what is claimed and desired protected byLetters Patent is set forth in the appended claims.

1. A computer-implemented method executed on a processor for reducingdata loading overhead of middleware to facilitate direct data loading toa database management system (DBMS), the method comprising: while dataloading: sending external-writes to a memory-based DBMS if correspondinginternal-writes are for vertices; exporting all the external-writes to adisk-based DBMS as an export file; and sending an external-read forvertices to an in-memory DBMS if the middleware requests data; and atthe end of data loading: generating files for raw data of the disk-basedDBMS from the export file; and sending the generated raw files to thedisk-based DBMS.
 2. The method of claim 1, wherein, while data loading,the sending of the external-writes step further includes exporting theinternal-writes as a recovery file.
 3. The method of claim 2, wherein,at a start of data loading, if the recovery file exists: generating theexternal-writes to the memory-based DBMS from the recovery file; andsending the generated external-writes to the memory-based DBMS.
 4. Themethod of claim 1, wherein, while data loading, the sending of theexternal-read step further includes sending the external-read to thedisk-based DBMS if the in-memory DBMS does not fetch any.
 5. The methodof claim 1, wherein an export extension of the middleware supports bothdisk-based DBMS and memory-based DBMS.
 6. The method of claim 1, whereinan export extension of the middleware processes the internal-writes andinternal reads.
 7. The method of claim 1, wherein, if internal-writesare not for vertices, appending the internal-writes to the export file.8. A non-transitory computer-readable storage medium comprising acomputer-readable program executed on a processor for reducing dataloading overhead of middleware to facilitate direct data loading to adatabase management system (DBMS), wherein the computer-readable programwhen executed on the processor causes a computer to perform the stepsof: while data loading: sending external-writes to a memory-based DBMSif corresponding internal-writes are for vertices; exporting all theexternal-writes to a disk-based DBMS as an export file; and sending anexternal-read for vertices to an in-memory DBMS if the middlewarerequests data; and at the end of data loading: generating files for rawdata of the disk-based DBMS from the export file; and sending thegenerated raw files to the disk-based DBMS.
 9. The non-transitorycomputer-readable storage medium of claim 8, wherein, while dataloading, the sending of the external-writes step further includesexporting the internal-writes as a recovery file.
 10. The non-transitorycomputer-readable storage medium of claim 9, wherein, at a start of dataloading, if the recovery file exists: generating the external-writes tothe memory-based DBMS from the recovery file; and sending the generatedexternal-writes to the memory-based DBMS.
 11. The non-transitorycomputer-readable storage medium of claim 8, wherein, while dataloading, the sending of the external-read step further includes sendingthe external-read to the disk-based DBMS if the in-memory DBMS does notfetch any.
 12. The non-transitory computer-readable storage medium ofclaim 8, wherein an export extension of the middleware supports bothdisk-based DBMS and memory-based DBMS.
 13. The non-transitorycomputer-readable storage medium of claim 8, wherein an export extensionof the middleware processes the internal-writes and internal reads. 14.The non-transitory computer-readable storage medium of claim 8, wherein,if internal-writes are not for vertices, appending the internal-writesto the export file.
 15. A system for reducing data loading overhead ofmiddleware to facilitate direct data loading to a database managementsystem (DBMS), the system comprising: a memory; and one or moreprocessors in communication with the memory configured to: while dataloading: send external-writes to a memory-based DBMS if correspondinginternal-writes are for vertices; export all the external-writes to adisk-based DBMS as an export file; and send an external-read forvertices to an in-memory DBMS if the middleware requests data; and atthe end of data loading: generate files for raw data of the disk-basedDBMS from the export file; and send the generated raw files to thedisk-based DBMS.
 16. The system of claim 15, wherein, while dataloading, the sending of the external-writes step further includesexporting the internal-writes as a recovery file.
 17. The system ofclaim 16, wherein, at a start of data loading, if the recovery fileexists: generating the external-writes to the memory-based DBMS from therecovery file; and sending the generated external-writes to thememory-based DBMS.
 18. The system of claim 15, wherein, while dataloading, the sending of the external-read step further includes sendingthe external-read to the disk-based DBMS if the in-memory DBMS does notfetch any.
 19. The system of claim 15, wherein an export extension ofthe middleware supports both disk-based DBMS and memory-based DBMS. 20.The system of claim 15, wherein an export extension of the middlewareprocesses the internal-writes and internal reads.
 21. The system ofclaim 15, wherein, if internal-writes are not for vertices, appendingthe internal-writes to the export file.
 22. A computer-implementedmethod executed on a processor for reducing data loading overhead ofmiddleware to facilitate direct data loading to a database managementsystem (DBMS), the method comprising: receiving an internal-write in anexport extension of the middleware; determining whether theinternal-write is for vertices; sending the internal-write to anin-memory DBMS when the internal write is for vertices; and appendingthe internal-write to a recovery file.
 23. The method of claim 22,wherein, if the internal-write is not for vertices, appending theinternal-write to an export file.
 24. A computer-implemented methodexecuted on a processor for reducing data loading overhead of middlewareto facilitate direct data loading to a database management system(DBMS), the method comprising: receiving an internal-read in an exportextension of the middleware; sending the internal-read to an in-memoryDBMS to receive a result; determining whether the result includes arecord; and if the result is free of a record, sending the internal-readto a disk-based DBMS.
 25. The method of claim 24, wherein, if the resultincludes a record, the result is returned to a caller.